DOCUMENT RESUME
ED 270 861
EA 018 541
AUTHOR
TITLE
INSTITUTION
REPORT NO
PUB DATE
NOTE
PUB TYPE
EDRS PRICE
DESCRIPTORS
IDENTIFIERS
ABSTRACT
Lawton, S. B.; And Others
Development and Use of Performance Appraisal of
Certificated Education Staff in Ontario Schoo"
Boards. Volume IV: Non-Technical Report.
Professionalism >n Schools Series.
Ontario Dept. of Education, Toronto.
ISBN-0-7729-1074-2
86
49p.; For Volume I, see EA 018 540. Volumes II and
III available only on microfiche from The Ontario
Institute for Studies in Education, 252 Bloor Street
West, Toronto, Ontario M5S 1V6 Canada. ISBN is for
the four-volume set.
Reports - Research/Technical (143)
MF01/PC02 Plus Postage.
Administrator Evaluation; *Educati onal
Administration; Educational Policy; Elementary
Secondary Education; ^Evaluation Criteria; Evaluation
Methods; ^Evaluation Problems; Evaluators; Foreign
Countries; Formative Evaluation; Management by
Objectives; Organizational Objectives; Organizational
Theories; Performance Factors ; ^Personnel Evaluation;
Standards; Summative Evaluation; *Teacher Evaluation;
Teaching (Occupation)
Ontario
This report, the fourth and final volume of a series,
examines in a summary fashion the results from a massive study of
performance appraisal policies and procedures currently used in
Ontario school boards. The study, a two-year undertaking, focused on
appiaisal practices for certified educational personnel in Ontario:
teachers, principals, consultants, superintendents, and directors.
Answers to four basic questions were sought: (1) What types of
performance appraisal policies have been adopted by Ontario school
boards? (2) To what extent have these policies been implemented? (3)
What types of appraisal practices are most effective? and (4) what
processes have school boards used to develop and implement their
performance appraisal policies? Based on a conceptual model relating
performance appraisal to organizational goals, the discussion focuses
on the following aspects of performance appraisal policy:
preparation data collection, reporting and followup, evolution of
policy, impact, and effectiveness. A summary lists 11 important
findings from the study, and the conclusion discusses policy
implications of the study as a whole. (TE)
*******************
* Reproductions supplied by EDRS are the best that can be made *
* from the original document. *
******************************************************************* ****
ERLC
DEVELOPMENT AND USE OF
PERFORMANCE APPRAISAL
OF CERTIFICATED
EDUCATION STAFF IN
ONTARIO SCHOOL BOARDS
Volume iV: Non-Technical Report
Professionalism in Schools Series
S. B. LAWTON, Principal Investigator
E.S. HICKCOX
K.A. LEITHWOOD
D.F. MUSELLA
This research project was funded under contract
by the Ministry of Education, Ontario.
This study reflects the views of the authors and not
necessarily those of the Ministry of Education.
The Honourable Sean Conway, Minister
© QUEEN'S PRINTER FOR ONTARIO i 1986
Order Information:
Publicationi Salei
The Ontario Institute
for Studiei in Education
252 Bloor Street Veit
Toronto, Ontario M5S 1V6
Will invoice on orders over $30.00.
Other orders must be accompanied by a
cheque or money order payable to
O.I.S.E.
Publications Service!
880 Bay Street, 5th Floor
Toronto, Ontario M7A 1N8
(416) 965-6015
(Toll Free) 1-800-268-7540
(Toll Free from area code 807)
Ask operator for Zenith 67200
Order must be accompanied by a cheque
or money order payable to the
Treasurer of Ontario
Vols. II and III are available only on microfiche
0N0 2845
Canadian Cataloguing in Publication Data
Main entry under title:
The Development and use of performance
appraisal of certificated education staff in
Ontario school boards
Contents: v. 1. Technical report. Overview — v. 2.
Technical report. Appendix A, Appendix B — v. 3.
Technical report. Appendix C — v. 4. Non- technical
report.
Volumes 2 and 3 published in microfiche only.
ISBN 0-7729-1074-X (4 v. set)
1. Teachers — Ontario — Rating of. I. Lawton,
Stephen B. II. Ontario. Ministry of Education.
LB2838.D48 1986 371 . 1 1 4409713 C86-099626-3
ERLC
CONTENTS
FIGURES ANO TABLES iv
ACKNOWLEDGMENTS v
I: INTRODUCTION 1
II: METHOOOLOGY 6
III: OEMOGRAPHIC PROFILE 9
IV: OEFINITIONS 13
V: PREPARATION FOR APPRAISAL 16
VI: OATA COLLECTION 23
VII: IMPACT 29
VIII: EVOLUTION OF POLICY 33
IX: SYSTEM OESIGN ANO RESULTS 34
X: SUMMARY ANO CONCLUSIONS 38
ERLC
FIGURES AND TABLES
Figure 1: Systems Model for Performance Appraisal 2
Table 1: Age Distribution by Role of Respondent 9
2: Highest Academic Qualification by Role
of Respondent 10
3: Mean Number of Staff Placed Under Review
8etween September 1981 and May 1983 by Type of 8oard 11
4: Mean Number of Staff Dismissed for Unsatisfactory
Performance Between September 1981 and May 1983 by
Type of 8oard 12
5: Degree of Improvement in Teachers' Performance After
Evaluation as Perceived by Teachers and Superintendents 30
6: Degree of Improvement in Principals' Performance After
Evaluation as Perceived by Principals, Teachers, and
Directors 31
iv
6
ACKNOWLEDGMENTS
The co-operation and efforts of a great many people are required in order to realize a major
research project such as this study of performance appraisal practices in Ontario school
boards. Both their m 1 * jers and the necessity for confidentiality, however, prevent our
naming those to whom we are most grateful: the five thousand teachers, principals,
consultants, superintendents, directors, and trustees in thirty school boards who responded
to our questionnaires; the administrators in over one hundrec school board* who responded to
our request for copies of their current evaluation policies; and especially the nine school
boards that served as sites for case studies (including the board that was used to "pilot
test" the research procedures).
Members of the Advisory Group designated by the Ministry of Education can be thanked by
name: Bruce Archer, Noel Clark, Bert Chalmers, Sandra Gaskell, Robert Lefebvre, and Mike
Prokopich. This group acted as a sounding board and helped to keep the research team on
track throughout the two years required for the study. T s able guidance of Peter
Nightingale of the Research and Information Branch of the Ministry of Education, who acted as
the supervisory officer on the project and as Chair of the Advisory Group, was particularly
appreciated.
The research itself involved many individuals. Central among these was Mary Stager,
full-time research office? on the project, who co-ordinated all research activities,
including the surveys, case studies, and preparation of reports, conducted interviews for
case studies, and prepared statistical analyses of questionnaire data reported in Volume Two
of the Technical Report. In addition, she is author of the chapter on evaluation policies in
Volume One of the Technical Report. She was ably assisted on a part-time basis by two other
research officers: Monique Belanger, who translated questionnaires into French and conducted
interviews with franco-Ontarian educators, and Nancy Watson, who conducted many of the
interviews on which the case studies were based.
Two graduate assistants, provided by the Department of Educational Administration, OISE,
undertook special analyses. David To analysed data from the screening questionnaire, and
Susan Sydor synthesized the open-ended responses on questionnaires from directors and
superintendents.
v
Secretarial services for the project were provided by: Elaine Tanenzapf, who helped to
organize and administer the screening survey; Loukritia Prattas, who transcribed many of the
interview notes; Monique Nicole, who typed the French- language questionnaires; Jan Swanson,
who assisted in typing the final manuscript; and Elizabeth Fear, who served as the project
secretary-- transcribing notes, typing interim reports, and preparing the manuscripts for the
final reports. As well, assisted by Paul Chau and John Chen, two secondary school students
employed part time during the summer of 1983, she was responsible for keying much of the data
and editing the data files in preparation for analysis. The quality of the work evident in
this report reflects her commitment to excellence, a commitment that deserves both our
respect and appreciation.
ERLC
S
vi
I. Introduction
This report examines in a summary fashion results from a massive study of performance
appraisal policies and procedures currently used in Ontario school boards. The study, a
two-year undertaking, focussed on appraisal practices for certificated educational personnel
in Ontario, namely, teachers, principals, consultants, superintendents, and directors.
Answers to four basic questions were sought: What types of performance appraisal
policies have been adopted by Ontario school boards? To what extent have these policies been
implemented? What types of appraisal practice are most effective? And, what processes have
school boards used to develop and implement their performance appraisal policies? In the
following pages, we present information which attempts to answer these questions or at least
throws light on some of the underlying issues.
To provide guidance for developing specific questions to address to respondents and to
assist in the analysis and presentation of information, we worked from a conceptual model.
(See Figure 1.) This model assumes that performance appraisal systems grow out of
organizational goals and objectives.
i
Figure 1
Systems Model for Performance Appraisal
s s 1
| Organizational |
/ I Goals and Objectives | \
/ I I \
/
\
\
/
2 /
| Organization |
J Structure I
Purpose of |
Appraisal |
! Job
i Descriptions
1 \
Criteria
Contracts
A I
1/
Individual
Objectives
Job
Performance
Data Collection
Type
Source
Method
T\
/ \
/ \
/ \
/ \
/ \
/ m
Follow- |
Through |
/
10
Performance
Review
Analysis of Data
Decision (Type &
Method)
Method of Feedback
PREPARATION
DATA
COLLECTION
REPORTING
AND
FOLLOW-UP
12
| Assessment of |
, Appraisal System |
IMPACT
EVOLUTION OF POLICY
10
From these organizational goals, purposes for appraisal of personnel are developed.
Ideally, these purposes are written down and criteria are developed to determine if the
purposes have been achieved. Information is collected and a review or evaluation takes
place. In an ideal system, also, there is feedback, called "follow-through" in this study.
Personnel evaluation or appraisal takes place within an organizational setting, at least
in Ontario schools; thus, any examination of appraisal has to recognize that organizational
structure, job descriptions, and contracts have an interactive relationship with the
appraisal system itself. Further, since individuals bring their own agendas into the job
situation, the subsequent job performance is therefore not just a matter of following a job
description or meeting organizational goals. As well, in an ideal situation, the appraisal
system itself must be subject to regular assessment.
In one sense, the study being reported here is an attempt to determine the degree to
which appraisal systems in Ontario schools are congruent with the ideal model presented in
Figure 1. Our fundamental purpose, however, is not to prescribe but, rather, to describe,
and to raise questions worthy of consideration by policy-makers. Thus, we work from an ideal
model with a limited goal. We use it to impose order on our information with the hope that
some important understandings will emerge from the extremely complicated set of human
interactions that comprise performance appraisal systems in Ontario schools. One of the
major reasons for the difficulties associated with personnel evaluation, in fact, is that it
involves intensive human interaction, with the possibility of an adverse judgment abou 1 " an
individual's performance that could result in a damaged career.
Three distinct research strategies were employed to collect the information. (The next
chapter analyses the methodology in more detail.)
1. Analysis of Policies and Procedures . School boards in the province were asked to
provide all written information they had developed relative to performance appraisal for
all categories of professionals. This information was subjected to document analysis.
2. Survey . A sample of thirty school boards was selected, and professional staff and
trustees in these boards were asked to respond to an extensive questionnai re. There
were 4092 teacher respondents to this survey, and close to a thousand respondents in
other staff categories. Included in the sample were two boards with large francophone
populations.
3. Case Studies . Eight school boards were selected for intensive on-site interviewing.
These visits enabled us to examine in depth how people in schools feel about evaluation.
One of the boards had a large number of francophones, who were interviewed in French.
The product of these activities was a tremendous amount of information — probably, in
terms of quantity, the largest effort of this kind carried out anywhere.
3
There are three other reporting documents for this study, in addition to the present
one. The survey data are contained in an extensive technical report. The individual case
studies appear in a second technical report. The document analysis as well as a synthesis of
both the survey data and the case studies are contained in a third report. Finally, this
paper, less formal and shorter than the others, is labelled a "non-technical" report.
The framework for discussion, extracted principally from the conceptual model given in
Figure 1, is as follows:
erJc
1. Preparation . Preparation for performance appraisal includes four major aspects:
planning, purposes, criteria, and standards. Planning includes such matters as
notification of those being evaluated, and the holding of pre-evaluation conferences.
Purposes include the intended outcomes (e.g., whether or not a permanent contract should
be granted). Criteria, as used in this study, refer to indicator? that measure some
quality or behaviour; some might be quite complex, such as the style of classroom
management, while others might be straightforward, such as punctuality. Standards refer
to the level of expectations regarding criteria.
2 - Data Collection . This category includes both the sources and types of information
collected, who collects the information, and how much time is spent in collecting it.
For example, we reported how often teachers are observed for purposes of evaluation.
3 - Reporting and Fol low-Up . Included here are the nature of the report provided, its
destination, with whom it is shared, and any follow-up activities, such as plans for
action, that are developed.
4 - Evolution of Policy . We examine the process by which the policy was developed
(including who participated), the activities undertaken to implement it, reviews of the
policy, and the extent of specificity found within the policy. One important
distinction in policy evolution is between policies that separate administrative from
developmental purposes and those that employ the same procedures regardless of the
purpose.
5. Impact of Policy. This includes the degree of compliance with the policy, the extent of
effort expended in its implementation and administration, as well as the nature aid
degree of its impact.
6. Effectiveness . A discussion of factors involved in the effectiveness of evaluation
systems concludes the substantive section of the report.
While the topic of appraisal of professional educators has been an extremely
popular one for many years in Ontario and in other jurisdictions in Canada and the
United States, the issues appear to be in particularly sharp focus at trte present time
in Ontario. In addition to the general acceptance among educators and the public about
the desirability of better accountability, there are the facts of declining enrolments
and tough ecommic times. These realities have kept interest in evaluation issues at a
relative high level for a long time, and perhaps now there is more support than ever
before for coming to grips with some of the more difficult problems in the
implementation of effective evaluation practices.
One of the general findings from our study, in fact, is that, while a great deal is
known about what makes an effective set of appraisal policies and procedures, many
school systems in Ontario have not implemented such practices consistently.
5
II. Methodology
Full description and discussion of the methodologies employed in the study are to be found in
the technical reports. What follows are some pertinent aspects of the methodology.
Policies and Procedures
In the fall of 1982, we sent directors of all 187 Ontdrio school boards letters
requesting copies of policies, guidelines, and instruments relevant to their beard's
performance appraisal practices. In addition, we asked them to respond to a screening
questionnaire which asked for information about enrolment, adoption sequences of policies,
and similar information. Early response resulted in our decision to limit the requests to
the 77 public boards of education and the 49 Roman Catholic separate school boards in the
province. By January 1983, after follow-up efforts, we had received replies from all boards,
with the exception of two public and five separate boards.
While V\e amount of information provided by boards varied considerably, we sense that
what we have in hand is a complete set of written materials relative to performance appraisal
from virtually ell educational jurisdictions in Ontario.
Survey
We decided to obtain survey information through a sample of boards that had responded to
the request for information about policies and practices. In the end, we selected thirty
boards for the sample, based on traditional criteria including regional representation,
public and separate orientation, varying size, and variety in appraisal systems. Included in
these thirty boards were two boards with large francophone populations.
Having chosen the boards, we then engaged in a sophisticated process for determining wlio
the respondents would be within a board. Using the Directory of Education , 1982/83, we
selected schools within the sample boards on a random basis.
Using information from the conceptual framework, from other studies, and from the
experience of the investigators, we developed lengthy instruments for teachers, principals,
superintendents, directors, and trustees. These were field tested in a school board, and
subjected to critical analysis within the investigating team.
14
Questionnaires were sent tr, the sample boards. In elementary schools and high schools
of less than forty teachers, all teachers were sent the survey. All principals (with some
exceptions), all superintendents, and all directors, as well as a small number o* trustees
from each bo*rd, were asked to fill out instruments. With some exceptions, this process was
followed systematically, with the result that 5655 teachers, 1211 principals, 214
superintendents, 30 directors and 150 trustees received instruments. To the two boards with
1 arge French-speaking populations, questionnaires in French were sent to francophone
teachers, principals, and trustees, while English-language questionnaires were distributed to
ti.e directors and superintendents and to a sample of anglophone teachers and principals.
With respect to teachers, 4092 completed the English-language questionnaire, for a
return rate of 72.2 per cent; 139 completed French-language questionnaires, for a return rate
of 69.2 per cent. For the English-language boards, 33 per cent of the respondents wer3
public elementary teachers, 36 per cent were public secondary, and 31 per cent were separate
school .
Of the principals, 879 returned questionnaires, for a rate of 73 per cent. Fifty-one
per cent were "n elementary public schools, 37 per cent were from separate schools, and ±2
per cent *ere from public secondary schools.
Of the superintendents, 114, or 53 per cent, returned questionnaires, representing
twenty-five boards, thirteen public and twelve separate.
Twenty-six directors responded, a rate of 87 per cent, with fourteen being from public
boards and twelve from separate boards.
Considerable detail and discussion of all the processes involved in the data collection
are available. Our general feeling is that this samp ( e is representative of the province as
a whole in terms of the range and type of school board, and in terms of the categories of
personnel. A special effort was made to include French-language respondents. And the number
of respondents, particularly in the teacher and principal categories, lends a great measure
of confidence to the results.
Case Studies
Documentary analysis and survey techniques, while exhibiting great benefit in certain
aspects of inquiry, also have limitations. To provide a better range of information, eight
case studies were conducted in school boards which were part of the original thirty,
including one where French was the language ui instruction. The research was carried out by
multi-day visits to the school boards by two researchers, including one of the principal
investigators, who conducted in-depth interviews with all categories of personnel.
The data collected were analysed and presented in eight separate reports. One unique
aspect of this information is that it includes material about the appraisal of consultants,
material that had not been covered in the survey or the document review.
ERLC
7
In general, this study was treated as an exploratory one in which each stage built on
the previous stage. First, existing policies on the performance appraisal of teachers,
principals, superintendents, and directors we^e requested from all Ontario school boards.
Second, the content of these policies was analysed to develop a typology, which is the
fra r <*work described earlier. Third, a description of the frequency of different types of
policy elements was prepared. Fourth, a set of questionnaires was developed based upon the
typology; these were sent to schools in diverse settings with diverse kinds of appraisal
policies. Fifth, the statistical data were analysed and eight school boards were selected
for case studies, which probed the developmental processes that could not be adequately
captured in a questionnaire survey. Finally, reports were prepared describing the results.
lb
9
ERIC
III. Demographic Profile
"hat follows is a brief discussion of the characteristics of the respondents to the survey
.•istrument. This information is important to judge whether or not the respondents were
representative of the total population. (Our opinion is that they were.) Also, many of the
points are interesting in themselves in terms of attitudes toward appraisal.
Age
As one v id expect, the age of respondents tends to increase with the level of the
position. (See Table 1.) Not reflected in the table are differences between those in public
and separate schools. The latter are, on average, somewhat younger. For example, 23 per
cent of tht separate school teachers are thirtv years or under as compared with 12 per cent
of the public school teachers. Similarly, 10 per cent of the separate school principals are
thirty-five or under as compared with 2 per cent of the public elementary school principals.
Table 1
Age Distribution by Role of Respondent (%)
Age
in Years
Teachers
(n=4040)
Principals
(n=876)
Supt 1 s
(n=113)
Directors
(n=26)
Trustees
(n=75)
20 to 25
3.
2
0.0
0.
0
0.0
0.0
26 to 30
12.
5
0.3
0.
0
0.0
1.3
31 to 35
24.
7
4.5
2.
7
0.0
12.0
36 to 40
23
7
15.9
7.
1
3.8
10.7
41 to 45
15
0
33.7
19.
5
11.5
24.0
46 to 50
9
2
19.9
30.
1
26.9
14.7
51 to 55
6
9
18.2
23.
9
46.2
20.0
56 to 60
3
3
6.7
15.
0
7.7
6.7
61 to 65
1
4
0.9
1.
8
3.8
8.0
over 65
0
0
0.0
0.
0
0.0
2.7
ERIC
9
Orientation
Seventy per cent of the teacher respondents were classroom teachers, with the remainder
being in categories such as department head, vice principal with teaching responsibilities,
and counsellor with teaching responsibilities. Fifty per cent of the principals worked in
public elementary schools, 12 per cent in puolic secondary schools, and 37 per cent in
separate schools.
Academic Qualifications
Table 2 reports the academic qualifications of the respondents.
Table 2
Highest Academic Qualification by Role of Respondent (%)
Highest
Qual i f ication
Teachers
(n=4028)
Principals
(n=878)
Supt' s
(n=113)
Di rectors
(n=26)
Teachers' Col lege, 19.4
no degree
B. A. or B.Sc. 59.7
M.Ed. , M.A. , 12.7
M.Sc. , or M.B.A.
."d.D. or Ph.D. 0.4
Other 7.7
1.0
29.5
67.5
0.5
1.5
0.9
14.2
76.1
3.5
5.3
0.0
19.2
73.1
7.7
0.0
Most respondents without degrees were in thfc elementary schools. In general, secondary
school respondents held higher degrees. In contrast to the situation twenty or thirty years
ago, educators in all categories tend to have at least a B.A. or B.Sc. degree, and we can
predict an increase in the numbers with M.Ed. 's or equivalent in the next few years.
Sex
Overall, 40.3 per cent of the teachers in the sample were male. In public elementary
schools, the percentage was 26.8, in public secondary 65.7, and in separate 25.4. In
addition, 87.7 per cent of the principals were male, as were 96.5 per cent of the
superintendents , and al 1 directors in the sample. (At the time of the study, three
directors in all of the boards of the province were women.)
ERLC
16
10
Contracts
Of the teachers, 92.5 per cent held permanent contracts, as did 93.6 per cent of the
principals. At the higher levels, term contracts were more prevalent, with 15.9 per cent of
the superintendents and 23.1 per cent of the directors having them.
Under Review/Dismissal
Some data were col 1 ected from di rectors concerni ng staff who had been pi aced under
review or dismissed as a result of performance appraisals. Our results show that the numbers
of such staff are very low. It appears that, in a typical school board, between one and two
teachers are placed under review in a given year, while one principal may be placed under
review once in two years. Negative ratings of superintendents are virtually non-existent.
Dismissals as a result of unsatisfactory performar.-.e are still rarer, occurring in a typical
board about once per year for teachers and once in five to ten years for principals.
Appeals, grievances, and lawsuits as a result of performance appraisals appear to occur in a
typical board once every five to fifteen years. Tables 3 and 4 give the specific results for
under r e/iew and dismissals.
Table 3
Mean Number of Staff Placed Under Review
Between September 1981 and May 1983
by Type of Board
Staff Category
Public
Separate
Total
(n=14)
(n=12)
(n=26)
Teachers
5.67
4.42
5.04
Principals
1.77
0.60
1.26
Superintendents
0.17
0.00
0.10
Other
1.14
1.29
1.21
0
ERJC
IS
11
Table 4
Mean Number of Staff Dismissed for Unsatisfactory Performance
Between September 1981 and May 1983
by Type of Board
Publ ic Separate Total
Staff Category (n=14) (n=12) (n=26)
Teachers 3.75 1.75 2.75
Principals 0.58 0.00 0.33
Superi ntendents 0. 00 0. 00 0. 00
In general, performance appraisal is most common at the classroom level and least common
at the director's level. The data suggest also a trend toward more evaluation in recent
years.
20
ERIC
12
IV. Defi nitions
In the course of the document analysis and the case studies analysis, we were confronted with
the rather common problem of definitions. Since the terms used to describe various
approaches to evaluation differ so much from board to board, we think it important to try to
clarify the various terms. An obvious recommendation is that some standard terminology
should be developed. Since appraisal of personnel is such a sensitive matter, it would seem
extremely important that there be some common understandings of what the topic of discussion
is, even when there is disagreement about what the policies and practices should be. What
follows is an attempt to sort out some frequent points of confusion in definitions.
At least nineteen of the boards submitting documents, referred to two distinct types of
evaluation in their written policies. These distinctions tended to be found in the more
elaborate teacher evaluation policies, but they are mentioned also for other categories of
personnel. Further, in the interview stages of the study, similar distinctions were almost
always referred to in statements to the effect that, even though it was not written down, the
system was engaged in both "formative and summative" evaluation. We discuss four sets of
distinctions.
1. Format i ve/Sumroati ve . These two terms, which appear frequently in the academic
literature, are used in a number of policies and procedures, but even the terms
themselves take on different meanings with particular boards. For example, several
boards agreed that "formative" refers to an ongoing developmental process while
"summative" occurs after a defined period has elapsed, From one board:
Formative evaluation is an ongoing
developmental process directed toward improved
performance and job satisfaction.
Summative evaluation provides a clear
perception of 'he overall effectiveness of an
individual's performance. It suras up... all
aspects of performance observed over a
definite period of time.
Other boards, however, attach other meanings to the terms. For example, in one,
"formative" tends to mean "informal" and "summative" to mean "formal".
Formative : Formative assessment is the
ongoing, informal approach to the improvement
of instruction.
ERIC
2i
13
Summative : Summative
formal approach to
review.
assessment
an annual
is the more
performance
Another approach (similar to the distinction made later between "administrati ve" and
"non-administrative" processes) is as follows:
Formative reports identify strengths in the
teacher* s performance and areas for
improvement or further development.
Summative reports are comprehensive and
summarize observations of the teacher's total
performance for purposes of promotion,
selection, exchange, recognition or as
required.
2. Administrative and Non-Administrative . In several boards, there is a distinction
between an "administrative" process and a "non-administrative" process (variously called
"improvement," "instructional," or "professional growth"), and the two processes entail
separate procedures. For example:
Evaluation for Professional Growth will be
directly related to the professional
development and/or improvement of teachers
will involve each teacher in the process
within a three year cycle
Administrative Evaluation tends tc be
summative in nature and requires indication of
the evaluator's support or non-support for the
teacher is the responsibility of the principal
and vice-principal (and sometimes the
superintendent) is clearly separated from the
process of Evaluation for Professional Growth
will not involve department heads, assistant
heads or consultants as evaluators leading to
administrative decisions.
3. Supervision and Evaluation . Closely related to both the above terminological
distinctions is that made between "supervision" and "evaluation. " Several boards agree
that "supervision" refers to a process involving giving helpful support to a teacher
while "evaluation" involves making a judgment.
Supervision
Supervision is a process whose primary
function is the improvement of a limited
number of aspects of the teacher 1 s work.
The result will be a report designed to be
helpful with suggestions as to how the
teacher' s work can further improve in the
future .
Evaluation
Evaluation is the process of making a judgment
about the overall quality of a teacher's work.
° 2z
ERIC 14 ^
Its primary function is to assist in making
administrative decisions about the teacher 1 s
future.
4. Classroom and Comprehensive . Finally, a distinction between "classroom" and
"comprehensive" evaluation is made explicit in a few boards. For example:
Formal teacher evaluation is divided into two
parts: I. The evaluation of classroom
performance, and II . The comprehensive
evaluation of the teacher' s contribution to
the total educational needs of the child, the
school and the Board of Education.
There are several problems with the variety of definitions. Of minor importance,
perhaps, is the difficulty in comparing evaluation systems across boards and in establishing
any kind of province-wide understandings about the meaning of various types of evaluation.
Of greater importance, perhaps, is that teachers and other professional educators undergoing
evaluation are almost certain to be disturbed by the vagueness and ambiguity of both the
terms and the meanings attached to them.
In summary, it is evident both in Ontario and in the literat're that commonly there are
two categories of appraisal. Whatever the terminology and the variations in meaning, the
most accurate portrayal of the central intent of these two categories would be as follows:
1. Appraisal designed to bring about change
in performance
2. Appraisal designed to enable a judgment
to be made about the value of
performance .
2,1
ERIC is
V. Preparation for Appraisal
This chapter discusses preparation for appraisal under several broad topics including
planning, purposes, criteria, and standards. There is some suggestion that educators in the
province look on the preparatory aspects of appraisal, where they have been effectively
carried out, as the most rewarding part of the whole process. The act of planning and
determining purposes, in some measure, is what motivates participants to take the process
seriously, far more than classroom observation or final reports.
In a general way, more information was available relative to preparation for teacher
appraisal than for any other category. The least information available related to directors.
Planning
Preparation for the evaluation of a teacher may be as simple as a principal's informally
dropping in unannounced while class is in session to 'see how things are going" or as complex
as a series of school-wide sessions to discuss objectives and agree on the criteria for
evaluation.
For teachers, notification in person is most common, overall, and is most prevalent in
public elementary schools. Memoranda are most likely to be used in secondary schools and are
more likely to be used in separate than public elementary schools. Informal visits or
observations in the classroom are likely to begin the process in almost hair of the cases in
public and separate elementary schools.
Francophone teachers for the most part had similar experiences with notification of the
evaluation process, although there were more informal classroom visits by supervisors for
notification than was the case for anglophone teachers.
With respect to most principals, their last evaluation began with personal notification
of the impending review, followed by a request for materials and a pre-evaluation conference.
Specifically, 58 per cent of the principals were notified personally. French-language school
principals had an even higher percantage of personal notification, 67 per cent.
In a majority of cases (67 per cent), superintendents were notified of a coming
evaluation in a written request for a statement about projected objectives, activities, and
plans. For directors, the situation is extremely mixed, the notifications ranging from
ERLC
2l
16
memoranda, informal visits, and discussions by trustees, requests for information, and
notification in person. No clear pattern emerges.
An important aspect of most ideal models for teacher evaluation is a pre-evaluatiop
conference. In this study, 36 per cent of the teachers indicated that such a conference took
place with their evaluator; and there was not much variation among different types of
teacher. The average conference lasted about twenty minutes, although there was great
variation, ranging from a few minutes to more than an hour. The case study data reveal that
some principals place great stock in these conferences, plan them, keep records, and go over
things extensively with the teacher. Others hold them essentially on the run, using them to
set the time for classroom observations. In the written policies, 57 per cent of the boards
which had policies required a pre-conference, but very little was specified as to what should
occur during it or how it should be conducted.
A pre-conference seemed to be much more common for principals, with 70 per cent
reporting one. In comparison, only 35 per cent of the superintendents and about half of the
directors reported cne.
The setting of objectives, while mentioned in only thirty-eight policies relating to
teacher evaluation, was more common for other categories of personnel, especially principals.
When pre-conferences were held for any length of time that is for at least the average of
twenty minutes for teachers, time would likely be spent talking about objectives for the
classes to be observed. In the cases of principals, consultants, superintendents, and
directors, pre-conferences, when they were held, almost invariably involved discussion of
objectives. These objectives varied considerably from rather full statements related to the
school system's goals and objectives to very specific matters such as the number of visits a
school superintendent would make to schools in his or her jurisdiction.
Notification, pre-conference, and planning for evaluation are important elements almost
without exception in writings about evaluation. The data in this study indicate, however,
that neither the written policies nor the perceptions of respondents in most categories hold
the preparation as a major aspect of the process. An exception seems to be for principal
evaluation where quite a l?rge majority reported pre-conferencing.
Respondents in the interview p k .ase of the study, in general, spoke of the importance of
preparation for evaluation. A major difficulty, however, was workload. In one board, for
example, superintendents were responsible for evaluating principals and quite a large number
of teachers. They simply did not have the time, along with other pressures, to pay more than
1 ip service to the responsibil i ty. In another board, superintendents were expected to
evaluate principals every year, as well as being involved in the supervision of a certain
number of teachers. No matter how noble the cause, there seemed to be a built-in set of
barriers to carrying out this phase of the appraisal process with any degree of
effectiveness. As indicated, planning the evaluation of principals did seem to be more
effective in this aspect than planning for other categories. Also, in schools where there
were very few teachers, say eight or ten, and a full-time principal, the principals reported
ERIC
17
quite favourably on the appraisal experience, relative to teachers, especially their sense of
achievement in the preparatory phases.
Purposes
The questionnaires and interviews used in the study allowed many specific purposes for
appraisal to be identified. These have been classified as education (e.g., improved student
learning), administrative (e.g., principal transfers), and policy-oriented (e.g.,
implementing Bill 82) purposes.
Most teachers (71 per cent), both francophone and anglophone, indicated that the
purposes of their last evaluation had been clearly communicated. This is a positive finding,
but somewhat at variance with the findings in the previous section that not much time was
spent in preparation for evaluation. Nevertheless, the respondents did not have much
confusion in their own minds.
Teachers appear to believe that the primary purpose of the evaluation they most recently
received was to comply with policy. Seventy-six per cent indicated that this was the case,
although only 40 per cent thought that this should be the case. The ideal purposes,
according to teachers, are to decide on permanent contracts for probationary employees and tc
improve instruction. Both teachers and principals agree that teacher evaluation should not
be used for administrative purposes such as selecting teachers for transfer.
Little difference was found among public elementary, public secondary, and separate
school teachers, although francophone teachers were less likely to report that compliance
with policies was the purpose of their last evaluation (about 46 per cent).
From examination of the policies and from the interviews, it is clear first that most
policies indicate improvement of instruction or some variation is the prime purpose of
evaluation, from the point of view of policy-makers. The interviews, however, tended to
reflect a mood among teachers that evaluation was something laid on that had to be endured,
that, while it was not really a threat, it was also not a thing that weighed on minds, except
during the period once every three years or so when observations were to occur.
The situation was somewhat differen* for principals. While a large majority (74 per
cent) thought a prime actual purpose was to comply with policy, there was also strong support
for more developmental purposes, "involving the improvement of their on-the-job performance.
Sixty- three per cent, for example, felt a prime purpose was to "stimulate improved
administrative performance". Further, there was closer agreement than there was for teachers
as to what is the case and what ought to be the case relative to the purposes of evaluation.
The written policies themselves emphasize developmental purposes over administrative
purposes. Fewer boards make written statements about purposes for principal appraisal than
for teacher appraisal.
ERLC
18
The same tendency is more pronounced for superintendents and directors. The higher in
ths hierarchy, the less comment there is about the purposes of appraisal. It is only at the
director's level that statements concerning purpose, focus on making an administrative
judgment rather than on improving performance.
For superintendents, items receiving a large majority of respondent choice included
assessing the achievement of objecti/es, complying with board policy, identifying
a'Jmi ni strati ve weaknesses , and assessi ng the achi evement of the board 1 s ob jecti ves.
Examination of these and other items in the data indicate the perception, at least, that the
fundamental purpose ic developmental. The interview data tend to confirm this perception,
a 14 hough perhaps less clearly.
Purposes which directors felt were of prime importance for their appraisals were
assessing the achievement of their objectives, complying with board policy, identifying
administrative strengths, and assessing the achievement of the board's objectives. Trustee
judgments were along the same line. Again, most of these perceptions relate to developmental
aspects.
We should note, relatwe to purposes, that there is variation among various categories
cf position, with the higher positions reflecting more concern with administrative matters as
opposed to classroom matters. This would seem to be reasonable and appropriate. All
categories of respondent- indicated a strong perception that a main purpose of evaluation was
to comply with policy. While this response may be an artifact of the way the question was
asked, it probably should be noted by policy-makers. Complying with policy is not a
particularly fundamental kind of purpose, and if it is the main one teachers and others
perceive, then some fu ther efforts at communication might be in order.
Criteria
One of the central issues in the area of performance appraisal over many years has been
the problem of criteria, used in this study to mean the indicators by which performance
should be measured. This issue is coupled with the notion of standards, discussed in the
next section, which is used here to mean c he level of expectations regarding criteria. The
problem with criteria is at least twofold: first, there is a tendency to employ terms and
concepts which are extremely vague and ambiguous, terms such as climate and enthusiasm;
second, criteria are often used which have no basis in research or experience in terms of any
significant relationship with effective teaching or learning. For example, punctuality,
while certainly a desirable behaviour in many situations, does not seem to be directly
related to effective teaching in all situations. In our sample of teachers, more than 50 per
cent of the 4000 respondents thought that punctuality i,i:ght always to be a criterion used in
evaluation.
Criteria formed the largest sing!e component of many written teacher policies.
Sometimes these involved long lists of possible criteria, with explanations of what they
me int. In other documents, the terms were simply listed. Criteria were also listed or
er|c 3 '
embedded in job descriptions, wh*re thcoe *"isted, and sometimes in the objectives of the
smool board. A few boards indicated that those being evaluated should have some say in what
the criteria for performance should be.
The form in which criteria are stated varies tremendously among boards, ranging from
very brief statements (e.g., "interpersonal relationships", "planning and preparation") to
lengthy Msts of behaviours expected of a good teacher. Sometimes criteria are broad areas
to be assessed by means of performance indicators and descriptors; at other times, they are
very specific indicators of competence in certain areas.
In our survey instruments, we asked each group of respondents to indicate from a list of
about twenty-five possible criteria which ones were actually used and which ones ought to be
used. For the administrative categories, we grouped these criteria into clusters labeled
"criterion domains" in order to try to identify general tendencies.
For teachers, both criteria they perceived as being used and those they thought should
be used tended to relate to classroom processes, e.g., items such as techniques of
instruction, teacher/pupil relations, classroom management. Moving away from the process
variables involved in the classroom to criteria related to studen* performance or toward
broader criteria involved in educational activities outside the classroom, we find less
frequent use of and less support for the criteria. Further, there is reasonable congruence
between criteria teachers would like to see and those actually used, at least in terms of
perception. Teachers in French-language schools showed similar results except that their
percentages for many items are higher than those for the English-speaking teachers.
For principals, there were twenty-two items grouped under five domains as follows:
general (administrative perfo? mance, personnel management, etc.); routine administration
(budget, records, etc.); interpersonal relations (parents, teachers, etc.); improving school
effectiveness (program development and evaluation, innovative activities, etc.); and other.
Results for principals indicate that the perception of appropriate criteria covers an
extremely broad range of behaviour. Almost all items showed a response rate of about 50 per
cent for the 879 respondents. Further, other categories of respondent — teachers,
superintendents, and directors — tended to agree with the principals as tn what the ideal
combination of criteria for judging principal performance should be. There are some
discrepancies, most notably that principals and teachers generally are not so strong on the
importance of contribution to the community as are superintendents and directors. Senior
administrators seem somewhat more interested in innovative behaviour than are the principals
themselves.
The notion of criterion domain was not quite so effective for superintendents since
there was a tendency in policies to list skills rather than particular expected behaviours.
In the survey instruments, however, we did list categories of criteria under headings as
follows: routine administration; interpersonal relations; knowledge; skills; and other.
ERIC
2U
Again, as for principals, there was an extraordinary response in the positive direction
for most of the thirty items both for what was actually used and what should be used,
although the ideal received consistently greater positive response. There was some lack of
enthusiasm for criteria related to routine administration and for community activities.
Management skills received very high responses. Personality received one of the lowest
responses, with 39 per cent indicating it was used and 36 per cent indicating that it ought
to be used. There was an obviously large difference between separate school and public
school superintendents relative to contribution to religious education.
For directors, criterion domains were the same as for superintendents. Again, there
were numerous items which directors felt ought to be emphasized in evaluation, with most
interest being in management skills such as decision-making and human relations and less
interest being in routine administration. No one wants to be evaluated on roucine
administration, it seems.
We examined trustee perceptions of what the directors should be doing, and found fairly
consistent agreement. Both directors and trustees exhibited some sense that directors should
be focussing more on what goes on in schools and less on what goes on in the office.
It is difficult, and perhaps dangerous, to analyse too closely the meaning of the
findings relative to criteria. For example, if, as is the case in most boards, the policies
suggest that improvement is the intent of appraisal, it is difficult to imagine showing
improvement on thirty different criteria. And yet, respondents consistently indicate a great
array of skills and behaviours as essential criteria. In the case study interviews, there
was considerable probing in this area, but we did not come away with the sense that Ontario
educators feel any problem with the issues surrounding criteria.
Standards are the levels of performance used as the basis for judging the adequacy of a
person's performance.
In the written policies and procedures, none of the responding boards had references to
explicit standards, although there were frequent references to criteria. The exception is
that instruments usad in appraisal often had scales for particular criteria as well as a
global rating scale. Occasionally, the instruments Carried a full explanation of what the
standard to be used meant, either through definition or example; the more common format,
however, was to have a scale ranging from poor to excellent set opposite the criterion or
behaviour under examination.
In practice, however, as indicated in the case study boards, exolicit standards are
often used, sometimes set co-operatively by the appraiser and the appraisee, as in deciding
that for the next year a superintendent would visit twenty schools. It was unlikely that
these standards would have a quality dimension.
Stewards
ERIC
With regard to teacher perceptions of who sets the standards, 38 per cent felt standards
were set by board policy, 48 per cent that they were set by the evaluator, 6 per cent that
they were set by the person being evaluated, 10 per cent that they were set collaboratively,
and 29 per cent did not know.
Forty-three per cent of the principals indicated that standards were set co-operatively
and 17 per cent were not sure. Sixty per cent of the separate school principals thought
standards were set collaboratively.
The data from various sources do not reveal any specific standard operating for the
evaluation of superintendents and directors. A possible exception is that superintendents
may be judged against some agreed-on standards relative to specific objectives set for the
year, such as agreement to visit a certain number of schools. The impression is, however,
that these kinds of standards are not too significant. What is significant for
superintendents is some indication from the director that he or she is doing a good job; for
the director, it is more the action of the board in extending his or her contract for another
year or term.
ERLC
VI. Data Collection
This section considers information gathered about the data collection phase of appraisal for
the various categories of certified professional personnel in schools. The discussion
centres on types of information collected, who collects the information, the time spent in
collecting it, and reporting it.
For teachers, of course, the prime mode for data collection is classroom observation.
For principals, consultants, superintendents, and directors, the basic technique is
consultation with the individual being appraised.
Sources and Typ^s of Information
The written policies and procedures indicate that most appraisal practices involve
collecting information from the individual being appraised, although other sources are often
available such as teachers, department heads, parents, and students.
With regard to teachers, 76 per cent of them said that they themselves were involved in
providing information, and 88 per cent thought they ought always to be involved. Only 38 per
cent said the principal provided information, although 76 per cent thought the principal
should provide information. Somewhat surprisingly, 16 per cent of the teachers reported that
students provided information for evaluation and, while that figure is relatively low, we
also know from the interviews that evaluators, particularly principals, often chat informally
with students about teachers.
While observation is certainly the most common means for collecting information about
teachers (as 96 per cent reported), only /8 per cent of teachers thought observation should
always be used. Instead, many teachers seemed to prefer interviews as a means of collecting
information. There was reasonable consensus across categories of professionals as to what
sources of information should be used, namely, notes from observations of classroom
performance. Nearly half of the teachers reported that plan books were used as a source of
information but only 27 per cent thought this should be the case.
Self-evaluation is hardly ever used, according to teacher respondents, although there
was stronger sentiment that it ought to be used, especially from superintendents, with about
25 per cent of them indicating preference for this type of evaluation.
3i
23
For principals, the primary source of information was the principals themselves, and
principals felt overwhelmingly that this was the way it should be. Superintendents also
provided information (40 per cent of the principals reported this), followed by teachers,
parents, students, and others.
Relative to who ought to provide information, 48 per cent of teachers felt that they
should be heavily involved, while 32 per cent of the principals shared this feeling.
In a pattern quite different from that for teacher appraisal, information for principal
evaluation often was collected from a variety of sources in addition to the principal,
including school staff members, parents, and students. In one of the use study boards,
there was an elaborate and systematic effort to involve others, incl uding questionnaires
administered to teachers and later shared at a meeting between the teachers and the
principal. Also, the superintendent made it a point to contact parents and students about
the performance of the principal.
For superintendents, interviewing was the most common mode of data collection. Some
board documents required self-evaluation forms to be maintained. Generally, only the
superintendent and the director were involved in the appraisal. There was very little
evidence to suggest that principals, teachers, or others were involved, although
superintendents themselves thought that principals should be involved as well as other
superintendents. Generally, for superintendents, the information took the form of objectives
written by the superintendent and some assessment it the end of a time period as to whether
or not the objectives had been achieved.
For directors, who, as we have noted, have the least systematic evaluation process,
information was generally puvided by themselves to a committee of trustees. Actually, there
was more variety reported by directors in the types of information submitted, a majority
indicating that, in addition to the objectives, there were interviews by committee, oral
reports, and reports on achievement of objectives. Directors reported that they would like
more direction from trustees than they actually received. In one case study board, the
director indicated he was trying to persuade the trustees to initiate a systematic evaluation
system for him as they had done for all other categories of staff.
Collectors of Information
For all categories of professionals examined in this study, information was most often
col 1 ected by one person. For exampl e , in ei ghty-ni ne boards 1 pol i ci es and procedures
relative to teachers, the principal was designated as the primary evaluator in 89 per cent of
the cases. Superintendents were involved as primary evaluators about 20 per cent of the
time, probably involving probationary teachers being considered for permanent contract.
When duties for teacher eval uation are shared, they are shared by vice-princ ipals ,
superintendents, the individual being appraised (self-evaluation), and, to some extent in
secondary schools, department heads. This pattern for teachers is essentially the same as
ERLC
24
indicated in other studies conducted in Canada over the past fifteen years. It is the
principal who is the prime actor in collecting information, with superintendents rather
heavily involved.
For principals, there are really two basic patterns. First, the principal's
superintendent is most often specified as the collector of information or the evaluator.
Self-appraisal is also mentioned in many policies as appropriate for the principal. Second,
in thirteen boards the policies specified that a team of people would conduct the principal's
evaluation. In this kind of situation, it is generally a team of superintendents who hold
the responsibility. Typically, they would spend about a week in the school, talking to the
principal and teachers, examining the program, and the like. While this approach has been
criticized as an "inspectorial" approach , those systems which use it effectively find
generally positive acceptance. In this approach, there is usually a lengthy written report
covering all aspects of the school operation; in fact, in a sense, the basic assumption is
that the principal is to be evaluated on how well the school is doing in its total operation.
By contrast, when evaluation is the responsibility of one superintendent for a group of
principals, there is more likely to be a combination of objective setting, visits, and
consultations with the principal by the superintendent, along with some modified check list
to be filled out.
For superintendents, the director is the sole evaluator and collector of information.
We did not identify any system where a team approach was used. Typically, the superintendent
would meet with the director at the beginning of the year and again at the end to determine
whether or not objectives set at the beginning had been met. The approach in use generally
is not nearly so systematic as those for teachers and principals.
For the directors, it was most often the director who supplied information, when in fact
one could identify a genuine appraisal process. Trustees would be involved in interviewing
or discussions about what the director had done, but there was no evidence in most cases that
either trustees or other individuals within the system actually gathered information to be
used in appraisal of directors.
Except for the team approach used in some boards for appraisal of principals, the most
striking aspect of who collects the information is that it is generally done by one person.
As we note in the next section, this fact may be one of the key variables in whatever
problems there are with evaluation. For principals and superintendents, collecting
information about large numbers of people on a regular basis is perceived to use a great deal
of physical and psychic energy and, when combined with other responsibilities, appears to be
impossible to carry out with much effectiveness in many boards.
Time Spent in Collection
The amount of time spent in evaluation by evaluators was measured by asking how many
times teachers were observed and how often conferences took place. We have previously
25 33
indicated the incidence of pre-observation conferences (and their length) and the incidence
of planning conferences for administrators.
For teachers, board policies generally indicated how often evaluation was to take place,
but the frequency of observation was included in only about one-quarter of the policies.
Evaluation, as might be expected, is called for more frequently for probationary teachers
than for permanent contract teachers. The case studies indicated, also, that the amount of
time an individual supervisor spent evaluating an individual was not very great, a day or
less. The problem comes, as indicated in the previous sections, in the number of individuals
for whom a supervisor is responsible.
More than half of the teacher respondents report one or fewer observations per
evaluation. About 20 per cent report three or more observations per evaluation. In general,
evaluations occur in three-year cycles. Given human nature, we should not be surprised to
find that principals and superintendents report more observations of teachers per evaluation
than do teachers. Seventy-five per cent of the principals report more than three
observations per evaluation. All categories of respondents think there should be more
observations than there are, except that supervisors do not see how they can find any more
time. The results for French-language school respondents were approximately the same.
With respect to principals, only twenty-seven boards specified the frequency tor
evaluation, and only nine indicated the number of days for collecting data in the school as
part of the appraisal. Annual evaluation of principals is the most common pattern reported
i n pol icies.
About half ot the principals, and 67 per cent of the Fronch-language school principals,
reported that one day or less was spent in their evaluation. Principals, superintendents,
and directors felt that more time should be spent. For example, more than half of the
superintendents and directors felt Lhat five or more days should be spent.
Twenty-eight boards had sections of policies dealing with the frequency with which
appraisal of superintendents should occur. Most said that it should occur every year.
Responses from superintendents about how much time was involved in their own evaluation
is quite vague and inconsistent, Our reading is that directors and superintendents tend to
weave evaluation into their day-to-day routines, and they are in quite frequent contact. The
case studies indicate that there is usually time set aside at the beginning of the year for
superintendents to talk over with the director what is going to happen during the year, as
well as another more or less formal meeting at the end of the year.
Most directors reported very little time, less than a day, spent on their own
evaluation. Our general finding is that directors, with some exceptions, do not engage in
formal evaluation, although there may be a meeting with trustees, typically during the
summer recess, when ideas and plans for the coming year are discussed.
ERLC
34
26
The data reported here support one of our basic contentions — that many problems
associated with evaluation are related to time spent. In formal evaluation, et least, which
is what this study is about, not much time is spent relative to the perceived importance of
the activ.ly. A possible exception are some boards where considerable time and effort are
put into the evaluation of principals.
Reporting of Information
The policies and procedures relative to teachers generally specify that there should be
post-appraisal conferences and reports, and there is specification about the form the report
should take. As is generally the case, however, the higher U p one goes in the hierarchy, the
less specificity there is in the policies.
In actual practice, the case study data indicate a confusing variation in the kinds of
reports done, what was done with them, who did them, and the like. The general feeling is
that not much is done in most jurisdictions with most pe.-sonnel in reporting and follow-up.
The exception would be those very few individuals who are placed on review. In these cases,
there is a great deal of information collected for the record. The fact is, though, that the
emphasis in evaluation in the province is on observation, not on preparation or follow-up.
And the document analysis revealed that, following such observation in the several boards in
the province which had highly developed policies regarding follow-up and reporting for
teachers, there is typically a report made up, often a checklist or a form of some kind which
is shared with the teacher. This is usually done in the post-evaluation conference, if there
is one.
Only 14 per cent of the teachers, however, reported any kind of a plan resulting from
the report. And of those who indicated that a plan was developed for improvement, only about
half indicated that it was monitored in any way. Only a very small percentage reported any
positive rewards from a positive evaluation, and an even smaller, almost negligible number,
reported negative effects. Similar results were found from the French-language school
respondents.
Final conferences at the end of an evaluation cycle, as opposed to post-observation
conferences, did not occur very often (24 per cent of the teachers reported them), and they
averaged twenty-one minutes in length.
Where there were written reports, teachers reported almost universally that they
received copies, and about half thought copies were sent to the central office.
French- language school teachers reported similar results, although fewer (67 per cent)
reported receiving written copies for themselves.
If we consider the post-observation conference as the most common follow-up activity, we
note that most teachers (79 per cent) felt no threat at the conference. They felt that they
got good feedback and that the praise they received was sincere; 69 per cent "felt good"
after the conference.
27
With regard to principals, about half of the boards which had policies regarding
principal evaluation required a post-conference between the principal and the appraiser, but
this was a less frequent requirement than for teachers. We note that policies required a
pre-conference more frequently for principals than for teachers.
Principals almost universally reported that a conference with their superintendent was
held at the conclusion of the evaluation period. For those principals involved in team
evaluation, there would be more than one superintendent involved. These meetings averaged
about an hour in length. Principals had a generally favourable reaction to these
conferences. There was no variation in reporting procedures for various types of schools.
Only about one-quarter of the principals (more at the secondary level) reported that a
plan of action for improvement was developed. Of those that had plans, about 60 per cent
said that they were monitored and that there were professional development activities
designed to bring about improvement.
Relative to superintendents, policies that spoke to superintendent appraisal in general,
specified the director as the appraiser. A small number (eighteen) said there should be a
written report, and only eight said anything about what the report should look like. These
eight were split between anecdotal reports and ratings along a set of items. Only one board
specified any follow-up activities in its policies.
Superintendent appraisal, as reported earlier in this document, is not particularly
systematic in most school boards. Directors generally sit down with superintendents on a
more or less regular basis. But only 17 per cent of the superintendents reported any
particular follow-up plans resulting from the appraisal process. There were letters of
commendation for about one-quarter of the respondents. Only 1 per cent reported a merit-pay
increase as a result of appraisal. Only 6 per cent reported any negative feedback.
For directors, generally the appraisal process, if there was any kind at all, consisted
of meetings with trustees. In general, both directors' and trustees' perception of the
reporting and follow-up was similar to that for superintendents, although directors were less
likely to report feeling good at the end of the session. It was not threatening. Actually,
it is difficult to say much since only a small number of directors (actually, seven) said
anything at all about follow-up. In general, we must conclude that not much of lasting value
occurred.
While it would seem logical that, if improvement were the basic intent of appraisal at
all levels, plans for improvement and follow-up should be an integral part of the process.
We cannot say from the data reported that such is the case. While teachers and principals
generally experienced post-observation conferences or post-evaiuation conferences in which
reports of the evaluators were shared, there is little evidence that anything much resulted.
Certainly, there were few negative impressions. Most felt it was a pleasant experience.
Actual plans for improvement were not much in evidence. Further, at the superintendent and
director levels, the situation was almost entirely informal.
ERLC
28
VII. Impact
An important aspect of appraisal is the impact that it has on the system. We have chosen to
describe this impact in terms of compliance with the policies, effort made at carrying out
the policies, and the degree to which the intent of appraisal was carried out.
Compliance
All eight case study boards indicated high levels of compliance in implementing existing
policies as applied to most roles. Any shortfalls experienced tended to be in time-related
areas such as inability to oDserve the number of classes required. There was little
indication of disagreement with the elements of the existing policies.
Sixty-seven per cent of teachers felt that practice followed policy either approximately
or exactly. Most of the rest, however, were not certain, a finding corroborated in the
interviews. Similar findings are reported for all other categories of personnel.
In general, practice seems to follow policy quite well. Repeating a point made earlier
in this report, however, we note that many superintendents and principals who were
interviewed felt time pressures to an extent that they were unable to do what they would
consider to be a thorough job of evaluation.
Extent of Effort
Regular, although not always frequent, appraisal activity was characteristic of the
boards in the case studies, and this situation is reflected in the surveys. Most policies
call for a cyclical process, most typically for teachers an evaluation every three years, and
anywhere from one to five years for principals.
Professional development days in which appraisal is the topic for study are an indicator
of effort. About 30 per cent of the teachers reported having experienced workshops dealing
with evaluation, usually, however, at the time when a new policy was being instituted. Of
those who had attended workshops, more than half said they were "average" in quality. More
French- language school teachers (39 per cent) had participated in evaluation workshops.
About half of the principals surveyed and about 75 per cent of the superintendents
placed a high priority on implementing the evaluation policy.
ERLC
3/
29
Similar results hold for t^e evaluation of principals. Principals were positive about
the experience, and a large majority said they took evaluation seriously. Principals also
felt that their appraisers were skilful or very skilful (83 per cent), and the appraisers
said they placed a high priority on principal evaluation.
The results for superintendent appraisal were not quite so positive. Not so many found
their appraisers (usually the director) particularly skilful, and smaller numbers took the
whole process seriously. Similar findings hold for directors, although trustees said that
they placed a high priority on the evaluation of directors. The problem is that there is
little evidence to show that the appraisal of directors is done very effectively in actual
practice.
Degree of Impact
One of the most dramatic findings from the survey came from asking respondents whether
or not they had experienced any improvement in performance as a result of evaluation.
In general, we cannot find much evidence to show improvement in performance that would
come anywhere near matching the amount of collective effort put into evaluation. At best we
can note that there were very few negative comments, that is, only rarely did respondents
feel that appraisal had negatively affected their performance. But, at best, we can say only
that there was a mildly positive feeling.
Table 5 shows the perceived degree of improvement in performance as reported by teachers
and superintendents.
Table 5
Degree of Improvement in Teachers' Pe- formance
After Evaluation as Perceived
by Teachers and Super intendents (%)
Degree of Improvement
Teachers
(n=3158)
Superintendents
(n=100)
None at all
40.5
0.0
A small amount
43.6
14.0
A modest amount
13.3
43.0
A substantial amount
2.6
43.0
ERLC
30
Over 80 per cent of the teachers said they perceived little or no improvement as a
result of the appraisal process. By contrast, most superintendents reported that they could
see either a modest or substantial improvement in teacher performance. One could, in a
sense, argue either way. Further, other data are confusing. Respondents reported that the
goals of evaluation had been achieved, and in general they felt it was a fair process and
felt good about it. In the interviews, however, we tried to probe what improvements or
changes could be attributed to evaluation, and respondents essentially could not think of
any, with the exception of a very few individuals who could point to major help as a result
of the experience. Many interviewees noted that they normally sought assistance when they
neaded it from consultants, specialists, department heads, and their principals. They did
not wait three years for a formal evaluation to remedy the problem.
At the least, concern should be felt that so many teachers do not perceive improvement
in themselves as a result of appraisal.
The perceived degree of principals' improvement after appraisal is reported in Table 6.
Table 6
Degree of Improvement in Principals' Performance After
Evaluation As Perceived by Principals, Teachers, and Directors (%)
Degree of Improvement
Principals
(n=523)
Teachers
(n=982)
Directors
(n=16)
None at all
18.5
26.1
0.0
A small amount
51.8
13.0
12.5
A modest amount
24.7
9.9
50.0
A substantial amount
5.0
1.9
37.5
Don't know
49.1
3 b
ERIC 3i
Although principals perceive more improvement than do teachers as a result of
evaluation , a great majority (70 per cent) see only a small amount of improvement or no
improvement. Teachers in general were vague about whether or not principals had improved as
a result of evaluation, although those who did have a sense of it tended not to see much
improvement. Directors, on the contrary, could see a lot of improvement in principals as a
result of evaluation.
Again, one could see different points of view in these numbers. The interview data,
also, are ambiguous, although in general, more principals seemed to be positive about their
evaluation experience than did teachers.
Superintendents and directors, particularly directors, could not point to major
improvements as a result of appraisal; they were rather neutral abojt the experience.
ERIC 32
VIII. Evolution of Policy
School board policies generally develop over a number of years, usually in response to a
"felt need" that msy have a political or organizational basis. What follows is a brief
discussion of the development and implementation of these policies.
In general, teacher evaluation policies, and those for other categor. <s of personnel,
are developed by superintendents and principals, with some involvement by teachers. Teachers
generally supporter! this approach, but would like to be more involved in the process than
they are. Teachers also felt that federation involvement and approval would be a good thing.
For other categories of personnel — principals, superintendents, and directors — the
developmental patterns were essentially the same. What is evident is that, the higher up the
hierarchy, the less specificity there is in the policies, and the less attention paid to the
development of policies. Thus, written policies contain hardly anything about appraisals for
directors.
What seems to be lacking in this area is very much attention to revision of policies, or
systematic updating. The impression from the case studies is that a system may, at any given
time, devote a tremendous effort to the development or revision of appraisal policies, but
then systematic or cyclical review will be stopped, the assumption bei>g that the job has
been <ione. Other characteristics of policy development include the greatest attention being
given to teacher policy. In fact, policies and procedures almost invariably are developed
first for teachers.
Some boards which appear to have policies and procedures approaching the ideal type,
took a long time to develop the materials. One case study board spen. several years in
intensive discussion and study, involving all categories of personnel, before adopting a
policy. This particular policy also provided for periodic review and updating. Our general
impression is that boards which lift policies and procedures from other places without full
and extensive consultation are likely to have a less than satisfactory appraisal sys*^
9
ERLC
4i
33
IX. System d esign and Results
While we make no claim to answering some basic questions about appraisal systems,
particularly with regard to expected outcomes, the elements of "good" teaching, the
relationship of appraisal to student learning, and the like, we have attempted to relate some
findings of the study to some of the basic elements in our ideal model. Following is a
discussion cf some of these results. We are assuming that, when consistent patterns of
positive relationships occur, we can infer that the characteristics in question make for a
more suitable evaluation system. This analysis, reported in detail in the technical reports,
takes the school board as the unit of analysis.
We should emphasize the tentative nature of the correlational analysis. Its purpose is
only to provide hints that might be explored more fully. Correlations are statistical
relationships, and do not necessarily imply cause and effect.
Teachers
With regard to teachers, we examined correlations with two types of basic variable.
First, we correlated the various elements of the appraisal process with what we called
intervening variables. These were teacher satisfaction with the evaluation form, fairness of
the evaluator, fairness of procedures, skilfulness of the evaluator, how seriously the
teacher took the evaluation process, and how seriously the evaluator took the process. We
might label these variables the effects of the evaluation process on teachers. Our
assumption is that favourable responses on these variables would point toward a better
system.
The second set of items refers more directly to the results of the evaluation process.
These items included the judgment of the director of education as to whether or not the
system was effective, the extent of improved teaching performance as indicated by
superintendents, achievement of evaluation goals for the system, achievement of evaluation
goals for teachers, and extent of improvement in teacher performance as a result of the last
evaluation. These items might be termed end-result items.
Type of planning seeirs to be related to results. In particular, a personal meeting with
the individual being evaluated before the actual evaluation is correlated with teacher
satisfaction, while drop-in visits are negatively correlated. Pre-conferencing seems to be
important in systems where respondents are satisfied with the system.
ERLC
4<j
3*
A second important aspect of planning is the use of objectives in evaluation. Where the
setting of objectives is systematic and expected, teachers are more likely to report gains in
performance as a result of the evaluation process.
Relative to purposes and criteria, only two characteristics emerge as being positively
related to the outcomes of teacher evaluation. When the purposes are clearly itated in
advance to the teachers, there is a positive correlation with the intervening variables
(satisfaction with report form, perceived fairness of procedures, skill of evaluator, serious
attitude toward appraisal, achievement of personal goals, and improvement).
The second characteristic which correlates with these satisfaction measures is the
criterion domain of classroom performance. Since almost all teachers reported that classroom
performance was a criterion domain, this variable becomes important. Omitting classroom
performance criteria from evaluation would undermine the legitimacy of the evaluation process
as perceived by teachers. This relationship holds for the intervening variables, but there
is no significant relationship with end-result measures.
With respect to data collection, the involvement of students in providing information (a
relatively rare occurrence), was positively correlated with how well the system achieves its
goals, but not with teachers' perception that evaluation improved their performance.
Observation and interviews are related to both intervening and end-result variables, while
the use of questionnaires and documents did not exhibit consistent relationships.
Specific notes and suggestions about what the evaluator has seen in class have
consistent positive relationships, even though general notes and examination of lesson plans
show no relationships. No other types of information used seem related to the results of the
evaluation system.
No relationships were found between degree of principal involvement and the results of
appraisal of teachers. Some relationships were found with the involvement of
superintendents, the number of times teachers were observed, and the length of the
post-observation conferences.
With respect to reporting and follow-up, no characteristic exhibits a broad pattern of
relationships, although the existence of a final wrap-up conference has a strong correlation
with the amount of improvement reported. Report forms with ratings or summary marks have
negative correlations with the amount of improvement reported by teachers.
The development of a plan, even though such a thing was not reported very often, has a
strong relationship to the results of evaluation systems. It also has a strong relationship
with four intervening variables.
Involvement of teachers and principals in the evolution and implementation of policy had
a strong positive relationship with both intervening and end-result variables.
35 4,i
Principals
The process of analysis for principals was the same as for teachers, although the
variables were slightly different. The intervening variables were prin^.^al satisfaction
with the report form, fairness of the appraiser's judgment, fairness of procedures,
skilfulness of the appraiser, how seriously the principal took the appraisal process, and how
seriously the appraisei *ook the process.
For the end-result variaoles, the items were the effectiveness of the appraisal system
as judged by directors, extent of improvement in principals' performance as reported by
teachers, extent of improvement in performance as reported by principals, and extent of
improvement in principals' performance as reported by directors.
Relative to planning for appraisal, it is clear again that pre-conferencing between the
evaluator, generally the superintendent, and the principal is positively related to the
intervening variables, especially when the conference is a reasonable length, close to ar
hour. No such relationship is indicated for effectiveness variables.
For purposes and criteria, there are two correlations of note. First, when purposes
were clearly given there was a strong correlation wit.i all of the intervening variables.
Only one criterion domain, contribution to the board, was related to the end-result
variables.
With respect to data gathering, there is some evidence that involvement of teachers in
providing information about principals is a useful component of a principal appraisal system.
The use of one of the most popular types of information, goal packages, is not positively
correlated with any of the end-result variables. Other sources of information do not show
any strong patterns of relationship.
Use of an appraisal team is correlated with the seriousness with which principals take
the process and directors' perception of the amount of improvement that comes as a result of
principals' appraisal. Greater involvement of teachers in the process shows relationships to
several intervening variables.
Time, again, proves to be an important rharacteristic. The number of days spent
collecting information is positively related to how seriously principals take the process and
how seriously they perceive it is taLen by their appraisers. The length of the
post-conference, if one is held, appears to be still more important.
In the area of reporting, the form of the statement provided to principals does not
relate significantly to any of the intervening or end-result variables, although forms with
scale ratings of various activities are negatively correlated with how seriously the
evaluation process was looked upon by principals.
44
ERIC *
While the existence of an appeal process is not correlated with any of the variables,
development of a plan for improvement of principals' performance has three significant
correlations, including satisfactic with the report, fairness of the procedures, and amount
of improvement perceived by the principals.
In the category of evolution and implementation of policy, involvement of trustees in
principal evaluation, although rare, was positively correlated to the fairn^oS and
seriousness variables. Overall, no strong patterns show in this category.
It was not feasible to carry out the same kind of correlational analysis for
superintendents and directors, given the small number of boards with explicit policies for
these two categories.
45
ERIC 37
X. Summary and Conclusions
Summary
This study addresses four questions: What types of performance appraisal policies for
educational staff have been adopted by Ontario school boards? To what extent have these
policies been implemented? What types of appraisal practices are most effective? And, what
processes have school boards used to develop and implement thei r performance appraisal
pol icies?
To answer these questions, we requested existing policies on the performance appraisal
of teachers, principals, superintendents, and directors from all Ontario school
boards. We analysed the content of these policies to develop a typology, and prepared a
description of the frequency of different types of policy elements. We developed a set of
questionnaires based upon the typology, and sent them to schools in diverse settings with
diverse kinds of appraisal policies. After analysing the statistical data, we selected eight
school boards for case studies to provide data for a comparative analysis of the development,
implementation, and effectiveness of performance appraisal policies.
Different modes of analysis were used at different stages of the study. Percentage
distributions were prepared for all characteristics included within the typologies of
performance appraisal policies, revealing which practices were most widespread. Percentage
distributions, broker down by type of school (public elementary, public secondary, and
separate) were prepared for all questionnaire items; responses on a given item were then
compared for individuals in different roles and with different languages of instruction. For
teachers and principals, a correlational analysis was carried out to determine which
characteristics of appraisal systems had an impact on the attitudes £.nd behaviour of those
evaluated. Finally, individual case study reports were prepared, and a comparative analysis
of these written.
The typologies used for all roles included i*-e major categories: preparation for
evaluation; data collection; reporting and follow-through, evolution of policy; and impact of
policies and practice.
9
ERIC
38
Findings of Importance
1. As of November 1982, 73 per cent of all school boards in Ontario had adopted policies
for teacher evaluation, 53 per cent for principals, 42 per cent for superintendents, and
36 per cent for directors.
2. Policies tended to become less detailed for positions higher in the hierarchy;
objectives-based appraisals, though occurring at all levels, were more frequently used
for positions higher in the hierarchy.
3. The primary purpose of appraisal at all levels was developmental; appraisal for
administrative purposes was somewhat more common at the director's level.
4. Evaluation by a single individual is most common at all levels, except that of director,
though information was often collected from several sources. Specifying the length of
time required to collect information was common for teachers and principals, but rare
for superintendents and directors.
5. Requirements for written reports, and plans to follow up on appraisals are most common
for teachers, less common for principals and superintendents, and least common for
directors.
6. For the most part, practices followed policies quite closely; where differences were
reported, they often reflected a failure to maintain proper schedules, to do required
numbers of observations, and the like. Those who were evaluated were more likely to
report such discrepancies than were those who conducted the evaluations.
7. Typically, evaluation was reported to increase the sense of well being of staff members
and to provide them with a few ideas on how to improve. The most impact was reported by
superintendents and the least by teachers. Directors were the only group who did not
feel better as a result of appraisal.
8. Characteristics of appraisal systems associated with favourable results of teacher or
principal appraisals include the holding of pre-conferences, longer pre-conf erences,
clear communication of purposes of the appraisal, the use of general and specific notes,
reports under several headings, post-conferences, and the making of plans.
Characteristics associated with the results of appraisals included the use of student
absenteeism data, and appraisals that began with informal visits from the evaluator.
9. Appraisals of administrators, especially of principals and superintendents, are seen as
being more effective than the appraisal of teachers, even though policies for the latter
are more detailed. The burden of appraising large numbers of teachers may make the
relative effort devoted to each teacher's appraisal much less than that devoted to each
administrator's appraisal. m -n
ERIC
39
10. Most respondents would prefer that more sources of date bt ^sed in their appraisals;
self-evaluation questionnaires were approved by all groups, but rarely used in practice.
11. Criteria used in appraisals related to out-of-classroom activities, were not widely
supported among teachers, though separate school and francophone teachers were more
supportive of their use than were teachers in public boards of education. A similar
si tuati on , wi th regard to out-of-school acti vi ti es , appl i es to pri nci pal s .
Superintendents and directors believe more emphasis in their own appraisals should be
placed on knowledge of laws and regulations; directors believe more emphasis should be
placed on their knowledge of schools and programs.
Conclusions
It was not within the scope of this research to make policy recommendations. It is perhaps
appropriate, however, to discuss briefly some of the main impressions we have from having
lived with this vast amount of information for so many months.
While it is clear that most school boards in the province have formal appraisal systems,
it is not so clear that they are being used with any particular effectiveness. Certainly, if
the objective is to improve instruction, the recipients of the process do ...v sense any
particular improvement. We believe, however, that appraisal systems can be made effective
without too much disruption to current approaches. The neutrality that educators feel about
being evaluated can be translated into a positive force by such obvious measures as ensuring
that all categories of professional personnel have policies and procedures that include the
major elements we identified as being important for a system to yield positive results.
There needs to be constant attention to the training of all concerned in the various skills
associated with appraisal. The conference procedure, for example, is not equally well
carried out by everyone, but there are simple aspects of it which can be mastered with a
little attention.
While there is an enormous amount of effort put into evaluation by administrators in
many boards, we could not really say that the results are used to any great effect.
Personnel files are filled with thousands of reports that are never really used, once they
have been written. To help individual s improve, appraisals could be used to enlighten
decision-making about changes in board goals, objectives, and structures.
In this non-technical report, we have not discussed appraisal policies and procedures
for consultants. Some boards have developed policies relative to consultants, and we have
reported on some of these in the case study analysis. In general, though, this is a
neglected area, and the consultants also feel neglected. While there are problems with the
evaluation of consultants, given the nature of their work, there are ways to proceed which
could prove useful.
48
erJc 40
One of the interesting findings from the study is that appraisal, if it is done
intelligently, even with the present set-up, is a big factor in the sense of well being among
school people. Professional educators, along with everyone else, have a need to be noticed,
and it is this finding that suggests to us that continued efforts should be made to improve
appraisal systems in all boards.
Many boards in the province have devoted tremendous resources and efforts to developing
appraisal systems. Some have done very well in certain elements, but not so well in others.
There should be a process whereby information can be exchanged about practices that have
worked and about failures as well. Most of all, there should be avoidance of tendencies to
gloss over problems, or to assume that just having an appraisal system in place is enough. We
are convinced, after this study, that appraisal is a dynamic process, and with the proper
commitment on the part of all parties concerned, including the Ministry of Education, it
could become the vehicle by which genuine improvement in the educational delivery system in
the province is made. But some thoughtful changes will have to be made in most board
policies and practices.
4 11
Er|c 4i
Live Music Archive
Librivox Free Audio
Metropolitan Museum
Cleveland Museum of Art
Internet Arcade
Console Living Room
Books to Borrow
Open Library
TV News
Understanding 9/11